DryadInc: Reusing Work in Large-scale Computations
نویسندگان
چکیده
Many large-scale (cloud) computations operate on append-only, partitioned datasets. We present two incremental computation frameworks to reuse prior work in these circumstances: (1) reusing identical computations already performed on data partitions, and (2) computing just on the newly appended data and merging the new and previous results.
منابع مشابه
Slider: Incremental Sliding-Window Computations for Large-Scale Data Analysis
Sliding-window computations are widely used for data analysis in networked systems. Such computations can consume significant computational resources, particularly in live systems, where new data arrives continuously. This is because they typically require a complete re-computation over the full window of data every time the window slides. Therefore, sliding-window computations face important s...
متن کاملA Three-terms Conjugate Gradient Algorithm for Solving Large-Scale Systems of Nonlinear Equations
Nonlinear conjugate gradient method is well known in solving large-scale unconstrained optimization problems due to it’s low storage requirement and simple to implement. Research activities on it’s application to handle higher dimensional systems of nonlinear equations are just beginning. This paper presents a Threeterm Conjugate Gradient algorithm for solving Large-Scale systems of nonlinear e...
متن کاملTracing Patterns of Large-Scale Software Reuse
Much work has been done on software reuse “in the small” – objects, subroutines, and components. Relatively little work on very large-grained reuse is available in the literature. Such reuse might include reusing an entire automated teller machine (ATM) software and hardware system for banks across multiple continents; and reusing infrastructure across Internet, teller systems, and ATM systems,...
متن کاملScheduling Large-Scale Parallel Computations on Networks of Workstations
Workstation networks are an underutilized yet valuable resource for solving large-scale parallel problems. In this paper, we present \idle-initiated" techniques for eeciently scheduling large-scale parallel computations on workstation networks. By \idle-initiated," we mean that idle computers actively search out work to do rather than wait for work to be assigned. The idle-initiated scheduler o...
متن کاملLarge-eddy simulation of turbulent flow over an array of wall-mounted cubes submerged in an emulated atmospheric boundary-layer
Turbulent flow over an array of wall-mounted cubic obstacles has been numerically investigated using large-eddy simulation. The simulations have been performed using high-performance computations with local cluster systems. The array of cubes are fully submerged in a simulated deep rough-wall atmospheric boundary-layer with high turbulence intensity characteristics of environmental turbulent fl...
متن کامل